Strength of consensus
Overview
The Strength of Consensus Score is given on an index level and is based on the percentage of the Network members operating each control with weighting given based on the level alignment. The score is given as a decimal between 0 and 1. It is designed to show the level of consensus around control alignment within a peer group for each index. This score should not be confused with Network Signal Strength which is calculated on a control level without weighting.
Purpose
To provide a measure of consensus, and hence data dispersion, to help guide the insights taken from the data. A low Network Alignment score – in both absolute and relative terms – will be of more concern where the Strength of Consensus Score is high. Highly dispersed data means a lack of consensus; severely reducing the usefulness of the consensus in benchmarking and setting a baseline. Discovering a poor consensus could demonstrate a risk susceptibility collectively across the peer group in that index Highly dispersed data shows that organisations within the peer group are addressing a wider set of threats offering the opportunity for horizon scanning and spotting developing trends.
Calculation – Explanation
- Average alignment score for each customer for each CUBE control is averaged. This will give a score per control of between 0 and 1. The alignment score is worked out as per the Network Alignment score; giving higher weighting to direct matches over partial matches with a maximum possible score of 1 per CUBE control. The scores are:
- 1 for direct match
- 1 for a control marked as not required
- 0.75 for a partial match (up to a maximum of 1 for a single CUBE control where multiple customer controls with partial matches exist e.g. Many to One) • 0 for a missing control
- These averaged control scores are then averaged across every control in an index to provide the consensus score for the whole index.
Calculation – Worked Example
For CUBE Control 1 in Risk index α:
| Customer | Match Type | Score |
|---|---|---|
| A | No Match | 0 |
| B | Many to One | 1 as this would be 0.75 alone but, in combination with the control below, the score increases to the maximum of 1 |
| B | Many to One | 0 as maximum of 1 per CUBE control has already been given in the customer control above |
| C | Aligned | 1 |
| D | Partial Match | 0.75 |
Average: 0.6875
For CUBE Control 2 in Risk index α:
| Customer | Match Type | Score |
|---|---|---|
| A | Aligned | 1 |
| B | Aligned | 1 |
| C | Aligned | 1 |
| D | Partial Match | 0.75 |
Average: 0.9375
For index α (consisting of two controls), the Strength of Consensus Score = (0.6875+0.9375)/2 = 0.8125
Practical Use of Score
Different levels of score lead to different possible interpretations. Some suggestions are provided:
High Consensus Score:
- The peer group has developed a strong agreement of the key risks and key controls within the index.
- Changes to the peer index and network updates for indices with a high consensus score are potential indicators of:
- Improvement in the control design and development
- Growth in inherent risk concerns for this peer group
- Changes in residual risk concerns for this peer group
- A customer should
- Review and implement controls labelled as “Missing” or “Preliminary Missing”
- Audit controls labelled as “Unique to Organisation” to reduce control proliferation
- Consider adopting the Network definitions and other control attributes where controls are labelled as a “Partial Match”
Mid Range Consensus Score
- There are pockets and early indicators of consensus around the key risks and key controls within the per group for this index.
- Changes to the index score and Network updates should be monitored as part of horizon scanning activities. In particular, risk types with the most movement can indicate emerging risks or increasing risk focus, and controls with reducing Network Signal Strength scores from these indices can indicate them as non key controls.
- A customer should:
- Review, implement or adapt controls for which there is a high Network Signal Strength.
- Audit controls labelled as “Unique to Your Organisation” to reduce control proliferation where there are a number of alternative controls linked to the risk.
indices Not Driven by Consensus
- Scores are only provided for indices where the number of contributors is three or more. This is to ensure that you have a higher quality of insights whilst always protecting the privacy of customer data.
- These indices still contain risk and controls influenced by a wide variety of sources including peers and independent experts. These indices will become consensus driven as more contributors input data to the index.